
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 1 3- 1 450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



10/644,829 



08/21/2003 



Kiyohide Satoh 



5514 7590 02/25/2005 

fitzpatrick cella harper & SCINTO 

30 ROCKEFELLER PLAZA 
NEW YORK, NY 10112 



02355.000096. 



6174 



EXAMINER 



BETZ, BLAKE E 



ART UNIT 



PAPER NUMBER 



2672 



DATE MAILED: 02/25/2005 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 


Application No. 

10/644,829 


Applicant(s) 

SATOH, KIYOHIDE 


F Y ami nor 

Blake E. Betz 


Art Unit 

2672 





~ The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 



- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)D Responsive to communication(s) filed on . 

2a)D This action is FINAL. I 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) H Claim(s) 1-27 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) M Claim(s) 1-27 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) Q Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) [X] Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)E3 All b)D Some * c)Q None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) Notice of References Cited (PTO-892) 

2) dl Notice of Drafts person's Patent Drawing Review (PTO-948) 

3) ^ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 

Paper No(s)/Mail Date 1. 

U.S. Patent and Trademark Office — 

PTOL-326 (Rev. 1 -04) Office Action Summary Part of Paper No./Mail Date 1 0644829 



4) O Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) C] Notice of Informal Patent Application (PTO-152) 

6) □ Other: . 



Application/Control Number: 10/644,829 Page 2 

Art Unit: 2672 

DETAILED ACTION 
Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1,6, 9, 14, 15, 17, 18, 24, 26, and 27 are rejected under 35 U.S.C. 102(b) 
as being anticipated by U.S. Patent No. 5,765,561 to Chen et al. 

Chen et al. discloses the system and method as claimed in claims 1 and 24. 
Chen et al. describes a surgical targeting system that involves displaying an image 
created by intermixing a virtual image with a real image. Column 9, lines 38 - 43, 
describes aligning a real image taken from a video camera with a virtual image by 
holding the camera in a fixed position. "A first, and generally more preferable, 
technique involves holding the position of video camera 45 (and hence real image 55) 
constant and moving.the position of the "virtual object" or the "virtual camera" by means 
of apparatus 57 until the virtual image 50 is brought into registration with real image 55." 



Column 11, lines 7 - 



11, states, "More particularly, and looking now at FIG. 8, there is 
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shown a video camera 45, an anatomical structure 60, and a tracker system 65. 
Tracker system 65 comprises a tracker 70 which is attached to video camera 45, and a 
tracker base 75 which defines the coordinate system of the tracker system." Thus, 
Chen et al. teaches of an attitude sensor for measuring the attitude of the video camera. 
Lines 12-17 disclose matrix transformations from the patient to the camera, the 
camera to the tracker base, and the patient to the tracker base. Thus, calculation 
information is stored in the matrix transformations to calculate the attitude of the patient 
on the basis of an output from the tracker system. Column 6, lines 61 - 67, and column 
7, lines 1 - 5, describe the use of an imaging machine to provide two-dimensional 
images of a patient's anatomical structure and storing the images in a patient-specific 
database. Column 8, lines 10-46, discloses the placement of virtual planning markers 
in the patient-specific images by a physician. Column 9, lines 43 - 47, describes a 
detecting algorithm to perform a template matching process between planning markers 
of a virtual image and a target image. "This can be done automatically by having image 
generator 30 use a search algorithm to match the virtual image to the real image, in 
which case apparatus 57 includes computer hardware and software of the sort well 
known in the art to cause image generator 30 to work through a search algorithm to 
match the virtual image to the real image." Column 12, lines 26 - 37, discloses 
detecting the position of an anatomical structure in a real image by performing a data 
matching procedure to correlate sampled points with patient-specific images. "It is also 
anticipated that one could use a tracked surgical instrument 85 to determine the location 
of anatomical structure 60. This can be accomplished by using the tracked surgical 
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instrument 85 to engage known fiducial points on the anatomical structure. 
Alternatively, a tracked surgical instrument 85 can be used to sample multiple surface 
points located on the'anatomical structure and then use a data matching procedure to 
correlate the sampled points with either patient-specific database 10 or patient-specific 
3-D computer model 1 5." Column 1 1 , lines 1 8 - 29, describe an updating of the 
calculation information on the basis of a detected position of a marker in the patient 
image then calculating the attitude of the patient in relation to the tracker system on the 
basis of known values and the updated calculation information from the matrix 
transformations. "M.sub.CT is known from the tracker system. Furthermore, once the 
virtual image generated by image generator 30 has been placed in registration with the 
real image generated by camera 45, the virtual camera position will be known relative to 
the virtual anatomical structure, and hence the real camera position will be known 
relative to the real anatomical structure. Thus, real matrix M. sub. PC will also be known. 
In addition, since M.sub.CT and M.sub.PC are then both known, it is possible to solve 
for M.sub.PT. Accordingly, the position of anatomical structure 60 will then also be 
known within the relative coordinate system defined by the tracker system." 

Chen et al. discloses the device of claims 6 and 9. Column 11, lines 12-29, 
discloses calculation information in the form of matrix transformations for calculating the 
position and attitude bf the patient on the basis of the measured value and position 
information of the pick-up visual point of the camera in regard to the tracker system. "In 
this setting, M.sub.PC can be considered to represent the matrix transformation from 
the patient's anatomical structure 60 to camera 45; M.sub.CT can be considered to 
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represent the matrix transformation from camera 45 to tracker base 75; and M.sub.PT 
can be considered to represent the matrix transformation from anatomical structure 60 
to tracker base 75. 

M.sub.CT is known from the tracker system. Furthermore, once the virtual image 
generated by image generator 30 has been placed in registration with the real image 
generated by camera 45, the virtual camera position will be known relative to the virtual 
anatomical structure, and hence the real camera position will be known relative to the 
real anatomical structure. Thus, real matrix M. sub. PC will also be known. In addition, 
since M.sub.CT and M.sub.PC are then both known, it is possible to solve for M.sub.PT. 
Accordingly, the position of anatomical structure 60 will then also be known within the 
relative coordinate system defined by the tracker system." Therefore, the calculation 
information is updated once the virtual image is placed in registration with the real 
image generated by the camera and the matrix transformations are formed. 

Chen et al. discloses the device of claims 14 and 15. Column 12, lines 26 - 35, 
describes using known fiducial points on the anatomical structure of a patient for 
tracking. "It is also anticipated that one could use a tracked surgical instrument 85 to 
determine the location of anatomical structure 60. This can be accomplished by using 
the tracked surgical instrument 85 to engage known fiducial points on the anatomical 
structure. Alternatively, a tracked surgical instrument 85 can be used to sample multiple 
surface points located on the anatomical structure and then use a data matching 
procedure to correlate the sampled points with either patient-specific database 10 or 
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patient-specific 3-D computer model 15." Thus, the patient in the video camera image 
may have known fiducial points located on their anatomical structure in real space. 

Chen et al. discloses the device of claims 17 and 18. Column 10, lines 1-15, 
states, "Once virtual image 50 has been placed into proper registration with real image 
55, image generator 30, Video camera 45 and video mixing device 35 can be used to 
present the virtual and real images on display 40 in various presentation formats so as 
to facilitate a particular medical procedure. In particular, one can use image generator 
30, video camera 45 and video mixing device 35 to superimpose a virtual image 
(generated from patient-specific 3-D computer model 15) against a real image 
(generated by video camera 45), with image generator 30 being directed to modify the 
virtual image so as to expose one or more of the virtual planning markers 25 present in 
patient-specific 3-D computer model 15, whereby the anatomy highlighted by virtual 
planning markers 25 will be brought quickly to the attention of the physician." Thus, real 
image is a visual point of the visual camera. Additionally, column 9, lines 43 - 49, notes 
a search algorithm to ; match the virtual and real images. "This can be done 
automatically by having image generator 30 use a search algorithm to match the virtual 
image to the real image, in which case apparatus 57 includes computer hardware and 
software of the sort well known in the art to cause image generator 30 to work through a 
search algorithm to match the virtual image to the real image." 

Chen et al. discloses the program code of claim 26 and the storage medium of 
claim 27. Figure 10 shows an overall flow diagram of the invention of Chen. A program 
module is shown that runs an algorithm to align the real and virtual images. Thus, the 
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algorithm to be run is interpreted as program code for executing the image processing 
method. Column 9, lines 38 - 49, describes the use of the algorithm including computer 
hardware and software such that the program code is stored on the hardware medium. 

Claims 2, 19, 20, and 21 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent No. 5,765,561 to Chen et al. in view of U.S. Patent No. 
5,531,227 to Schneider. 

Chen et al. discloses the device of claim 2 except wherein said target image 
setting unit obtains a. predicted position of the index in the picked-up image employing 
said measured value and said calculation information stored in said storage unit and 
creates an image with a peripheral area around said prediction position in said picked- 
up image subjected to a rotational process on the basis of a rotational angle in a roll 
direction of said image pick-up device derived from said measured value. The invention 
of Schneider teaches of obtaining images in a pre-procedural time and segmenting 
them into sub-object^ that have defined boundaries, shapes, and positions within the 
overall image. Column 8, lines 36 - 52, describes segmenting the images while they 
are digitally analyzed to identify predefined fiducial markers by means of feature 
extraction, edge detection, region growing, boundary analysis, template matching, etc. 
Column 9, lines 4 - 10, states, "Particularly likely to be useful in the future are those 
statistical segmentation techniques that assign to each point a certain degree of 
probability as to whether or not it is a part of a given segmented object. That probability 
is based upon a variety of factors including pixel intensity and location with respect to 
other pixels of given qualities. Once probabilities of each pixel have been determined, 
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assessments can be made of the pixels as a group, and segmentation can be achieved 
with improved accuracy." Additionally, column 10, lines 4-15, teaches of rotating 
captured images so as to align the images, according to the fiducial markers, to a 
corresponding position orientation. It would have been obvious to one having ordinary 
skill in the art at the time the invention was made to modify the invention of Chen et al. 
to include segmenting the captured virtual images so as to include a peripheral area 
around a predicted position of the index. One would have been motivated to make such 
a modification to the invention of Chen so that upon performing template matching 
between virtual and real images, the speed of identifying marker matches is increased 
by bounding a search area within a smaller particular region of the overall image. It 
would have been further obvious to one having ordinary skill in the art at the time the 
invention was made to modify the invention of Chen to include subjecting the captured 
images to rotational process. One would have been motivated to make such a 
modification to the invention of Chen so that while performing template matching 
between a virtual and a real image, the orientations of the images may be aligned so as 
to properly line up their fiducial markers. 

Chen et al. discloses the device of claim 19 except wherein the measurement 
object is a visual point of the observer, and optically transmitting the image in the real 
space through the display screen observed by the observer while displaying the image 
in the virtual space on the display screen. Column 7, lines 20 - 25 of Schneider, states, 
"A lead imager 18 is provided to obtain an image of object 10 along a chosen 
perspective or line of view. For example, if object 10 is a patient in an operating room, 
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lead imager 10 may be a video camera that obtains video images of the patient along 
the line of sight of the attending physician, such as a head-mounted video camera." 
Thus, the measurement object in Schneider is a visual point of the observer. It would 
have been obvious to one having ordinary skill in the art at the time the invention was 
made to modify the invention of Chen so that the measurement object is a visual point 
of the observer. During a medical procedure, it is important that a doctor quickly and 
easily obtains the orientation of planning markers on a patient with their perspective 
view of the patient. Thus, one would have been motivated to make such a modification 
to the invention of Chen so that during a medical procedure, the displayed view of a 
patient in a medical procedure would be oriented with the view of the doctor. Column 9, 
lines 38 - 49 of Chen, describes aligning a virtual image with a real image on the basis 
of the position of both images and the video camera so as to display the images 
simultaneously on a display screen. Column 10, lines 44 - 47 of Schneider, teaches of 
displaying a virtual image while optically transmitting a real image through the display. 
"Additionally, the transformed and sliced follow image can be projected onto a see- 
through display mounted in front of the physician's eyes so that it is effectively combined 
with the physician's direct view of the patient." Chen et al. teaches in column 10, lines 
59 - 63, that to perform image correspondence between a captured image and a virtual 
image requires a substantial amount of time, even with high speed hardware. 
Additionally, it is well known in the art that displaying an image on a graphics display 
requires an amount of hardware utilization. Thus, not displaying a real image by 
allowing the image to pass through a display and displaying a virtual image in alignment 
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with the real image will require less processing than if both images needed to be 
generated. It would have been obvious to one having ordinary skill in the art at the time 
the invention was made to modify the invention of Chen to optically transmit a real 
image through a display screen while displaying a virtual image in alignment with the 
real image as in Schneider. One would have been motivated to make such a 
modification to Chen so that less time would be required to generate the display by the 
hardware in the system. 

Chen et al. discloses the device of claims 20 and 21 , see description according 
to claim 1, except wherein a target image is created having a peripheral area around a 
predicted position in a picked-up image subjected to a rotational process on the basis of 
a rotational angle in a roll direction of said image pick-up device derived from said 
measured value. Chen et al. teaches in column 1 1 , lines 3 - 1 1 , of a tracker system 
attacked to a video camera such that the position of the camera will be predetermined 
by the tracker system. Schneider teaches of obtaining images in a pre-procedural time 
and segmenting them into sub-objects that have defined boundaries, shapes, and 
positions within the overall image. Column 8, lines 36 - 52, describes segmenting the 
images while they are digitally analyzed to identify predefined fiducial markers by 
means of feature extraction, edge detection, region growing, boundary analysis, 
template matching, etc. Column 9, lines 4-10, states, "Particularly likely to be useful in 
the future are those statistical segmentation techniques that assign to each point a 
certain degree of probability as to whether or not it is a part of a given segmented 
object. That probability is based upon a variety of factors including pixel intensity and 
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location with respect to other pixels of given qualities. Once probabilities of each pixel 
have been determined, assessments can be made of the pixels as a group, and 
segmentation can be. achieved with improved accuracy." Additionally, column 10, lines 
4-15, teaches of rotating captured images so as to align the images, according to the 
fiducial markers, to aicorresponding position orientation. Thus, the position of the area 
to be segmented is determined by a statistical segmentation technique while the video 
camera is at a predetermined position. It would have been obvious to one having 
ordinary skill in the art at the time the invention was made to modify the invention of 
Chen et al. to include segmenting the captured virtual images so as to include a 
peripheral area around a predicted position of the index. One would have been 
motivated to make such a modification to the invention of Chen so that upon performing 
template matching between virtual and real images, the speed of identifying marker 
matches is increased by bounding a search area within a smaller particular region of the 
overall image. It would have been further obvious to one having ordinary skill in the art 
at the time the invention was made to modify the invention of Chen to include subjecting 
the captured images to rotational process. One would have been motivated to make 
such a modification to the invention of Chen so that while performing template matching 
between a virtual and a real image, the orientations of the images may be aligned so as 
to properly line up their fiducial markers. 

Claims 3, 10, 11, and 12 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent No. 5,765,561 to Chen et al. in view of U.S. Patent No. 
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5,531,227 to Schneider as applied to claim 2 and further in view of U.S. Patent No. 
6,369,953 to Rallison et al. 

Chen and Schneider disclose the device of claim 3 except wherein said 
calculation information is the correction information to correct for an error in the 
measured value of attitude measured by said attitude sensor of the image pick-up visual 
point of said image pick-up device, and said calculation unit calculates attitude of said 
measurement object on the basis of the measured value and correction information. 
The invention of Rallison et al. teaches of tracking the position and attitude of an image 
generator mounted on the head of a person. Column 19, lines 1 1 - 26, teaches of the 
various types of tracker systems including magnetic and inertial sensor systems. 
Columns 21, 22, and 23 explain obtaining position and attitude orientation with the two 
systems. Column 24 ; , lines 37 - 67, and column 25, lines 23 - 62, describe correcting 
for an error in the measured values of attitude and position taken from the sensor 
systems. It would have been obvious to one having ordinary skill in the art at the time 
the invention was made to modify the invention of Chen et al. to include using an inertial 
sensor system as the tracker system for the video camera. One would have been 
motivated to make such a modification to Chen since inertial sensors such as rate gyros 
will have the advantage of being substantially immune to magnetic perturbations that 
may be present in a medical operating atmosphere. It would have been further obvious 
to one having ordinary skill in the art at the time the invention was made to correct for 
an error in the measured value of attitude information of the inertial sensors for use in 
the matrix transformation calculations of Chen. One would have been motivated to 
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make such a modification to the invention of Chen so that the calculated position and 
attitude information of the camera and patient remain accurate while performing a 
medical procedure. ■: 

Chen et al. discloses the device of claims 1 0, 1 1 , and 1 2. Column 1 1 , lines 1 2 - 
29, discloses calculation information in the form of matrix transformations for calculating 
the position and attitude of the patient on the basis of the measured value and position 
information of the pick-up visual point of the camera in regard to the tracker system. "In 
this setting, M.sub.PC can be considered to represent the matrix transformation from 
the patient's anatomical structure 60 to camera 45; M.sub.CT can be considered to 
represent the matrix transformation from camera 45 to tracker base 75; and M.sub.PT 
can be considered to represent the matrix transformation from anatomical structure 60 
to tracker base 75. 

M.sub.CT is known from the tracker system. Furthermore, once the virtual image 
generated by image generator 30 has been placed in registration with the real image 
generated by camera 45, the virtual camera position will be known relative to the virtual 
anatomical structure, and hence the real camera position will be known relative to the 
real anatomical structure. Thus, real matrix M.sub.PC will also be known. In addition, 
since M.sub.CT and M.sub.PC are then both known, it is possible to solve for M.sub.PT. 
Accordingly, the position of anatomical structure 60 will then also be known within the 
relative coordinate system defined by the tracker system." Therefore, the calculation 
information is updated once the virtual image is placed in registration with the real 
image generated by the camera and the matrix transformations are formed. 
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Additionally, column 12, lines 35 - 37, describes using a least squares fit to correlate 
the sampled points with the patient-specific images. Thus, a typical value is used that 
incorporates a dislocation value between the predicted position and the detected 
position of a plurality of markers from a real and virtual image. 

Claims 4, 5, 7, and 16 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent No. 5,765,561 to Chen et al. in view of U.S. Patent No. 
6,369,952 to Rallison et al. 

Chen et al. discloses the invention of claims 4 and 5 except wherein said 
calculation information is the correction information to correct for an error in the 
measured value of attitude measured by said attitude sensor and the position 
information of the image pick-up visual point of said image pick-up device, and said 
calculation unit calculates the position and attitude of said measurement object on the 
basis of the measured value, correction information, and the position information. The 
invention of Rallison et al. teaches of tracking the position and attitude of an image 
generator mounted on the head of a person. Column 19, lines 11 - 26, teaches of the 
various types of tracker systems including magnetic and inertial sensor systems. 
Columns 21, 22, and 23 explain obtaining position orientation with the two systems. 
Column 24, lines 37 1 67, and column 25, lines 23 - 62, describe correcting for an error 
in the measured values of attitude and position taken from the sensor systems. It would 
have been obvious to one having ordinary skill in the art at the time the invention was 
made to modify the invention of Chen et al. to include using an inertial sensor system as 
the tracker system for the video camera. One would have been motivated to make such 
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a modification to Chen since inertial sensors such as rate gyros will have the advantage 
of being substantially immune to magnetic perturbations that may be present in a 
medical operating atmosphere. It would have been further obvious to one having 
ordinary skill in the art at the time the invention was made to correct for an error in the 
measured value of attitude and position information of the inertial sensors for use in the 
matrix transformation calculations of Chen. One would have been motivated to make 
such a modification to the invention of Chen so that the calculated position and attitude 
information of the camera and patient remain accurate while performing a medical 
procedure. 

Chen and Rallison teach of the device of claim 7. Chen in view of Rallison 
teaches of updating the position information with regard to a series of matrix 
transformations in column 1 1 , lines 7 - 30. Additionally, column 9, lines 38 - 49, 
describe a search algorithm for automatically bringing a virtual image into registration 
with a real image. Column 10, lines 38 - 52, describes reestablishing the registration 
between a real image and a virtual image if the video camera position is moved. Thus, 
Chen and Rallison teach of updating position information. It would have been obvious 
to one having ordinary skill in the art at the time the invention was made to modify the 
invention of Chen and Rallison so that the position information is only updated in two 
directions and not in a depth direction if the action is so desired by a user while 
reestablishing a registration between a real image and a virtual image. One would have 
been motivated to make such a modification to the invention of Chen and Rallison so 
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that if an updated value in a depth direction is not required by a user, the computing 
system does not waste utilization time and resources calculating an unnecessary value. 

Chen and Rallison teach of the device of claim 16. Column 8, lines 10 - 30 of 
Chen, describes the index and virtual markers as being image features of interest to the 
physician in the patient-specific images. "More particularly, apparatus 20 preferably 
comprises computer hardware and software adapted to allow a physician to access one 
or more of the patient-specific 2-D images contained in patient-specific database 10 and 
present them for viewing, in the manner shown in FIG. 2. Then, using a mouse or other 
data entry device, the physician can place one or more virtual planning markers 25 into 
an accessed patient-specific 2-D image, in the manner shown in FIG. 3. These virtual 
planning markers 25 can consist of substantially any geometric form such as a point, a 
line, a circle, a plane, a path (either straight or curved), etc., and are positioned about 
anatomical structures of particular interest to the physician." 

Claim 8 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent No. 5,765,561 to Chen et al. in view of U.S. Patent No. 6,369,952 to Rallison et 
al. as applied to claim 3 and further in view of Fixsen et al. 

Chen and Rallison disclose the device of claim 8 except wherein the correction 
information is the information to correct for an error in the azimuth direction among the 
measured values of the attitude measured by the attitude sensor. In column 23, lines 
60 - 67, and column 24, lines 1 - 23, Rallison additionally teaches of sensor drift, thus 
causing an error in the measured values of roll, pitch, and yaw. Page 20, table 2 of 
Fixsen et al., and page 25, first paragraph, teaches of the interchangeability of the terms 
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yaw and azimuth in position measurements. Thus, it is inherent that the use of a 
gyroscopic tracking system in the invention of Chen in view of Rallison will accumulate 
an error in the measured directions, including the azimuth measurements. Therefore, 
the use of the yaw correction information in the matrix transformations in the invention 
of Chen in view of Rallison is equivalent to the use of the azimuth correction 
information. 

Claim 13 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent No. 5,765,561 to Chen et al. in view of U.S. Patent No. 6,369,952 to Rallison et 
al. as applied to claim 5 and further in view of U.S. Patent No. 5,531 ,227 to Schneider. 

Chen and Rallison teach of the device of claim 13 except wherein the updating 
unit updates the position information in three directions in the camera coordinate system 
of the video camera when two or more indices are detected in the detecting unit. 
Schneider teaches of determining the orientation, two-dimensional information, and 
size, three-dimensional information, regarding a captured and stored image via 
triangulation with multiple fiducial markers. Column 1 1 , lines 63 - 67, and column 1 2, 
lines 1 - 2, state, "For example, instead of determining the orientation of the object by 
determining the orientation of a single fiducial marker, as in the preferred embodiment, 
orientation and size information regarding the lead and follow images can be 
determined via triangulation by determining the relative position of the multiple fiducial 
markers as seen from a particular line of view." It would have been obvious to one 
having ordinary skill in the art at the time the invention was made to modify the device of 
Chen and Rallison to include updating the position information with regard to the matrix 
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transformations in three directions in the camera coordinate system when two or more 
indices are detected via triangulation as in Schneider. One would have been motivated 
to make such a modification to the invention of Chen and Rallison so that the position 
information as determined by the matrix transformations of Chen will provide the spatial 
relationship of an anatomical structure within the relative coordinate system defined by 
the tracker system. 

Claims 22 and 23 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over U.S. Patent No/5,765,561 to Chen et al. in view of U.S. Patent No. 6,369,952 to 
Rallison et al. and Fixsen et al. 

Chen et al. discloses the device of claims 22 and 23 except wherein the attitude 
sensor is constituted of a gyro sensor and the attitude sensor measures the attitude of 
an image pick-up point of the video camera in a state where there is an accumulated 
error in the measured value in the azimuth direction. Column 1 1 , lines 7-11, describes 
a tracker system attached to the video camera in the invention. Column 19, lines 1 1 - 
26 of Rallison, teaches of the various types of tracker systems including magnetic and 
inertial sensor systems. Column 20, lines 57 - 63, describes the use of a rate gyro in 
an inertial sensing system. It would have been obvious to one having ordinary skill in 
the art at the time the invention was made to modify the invention of Chen et al. to 
include using an inertial sensor system as the tracker system for the video camera. 
One would have been motivated to make such a modification to Chen since inertial 
sensors such as rate gyros will have the advantage of being substantially immune to 
magnetic perturbations, as stated in column 23, lines 37 - 40, that may be present in a 
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medical operating atmosphere. In column 23, lines 60 - 67, and column 24, lines 1 - 
23, Rallison additionally teaches of sensor drift, thus causing an error in the measured 
values of roll, pitch, and yaw. Page 20, table 2 of Fixsen et al., and page 25, first 
paragraph, teaches of the interchangeability of the terms yaw and azimuth in position 
measurements. Thus, it is inherent that the use of a gyroscopic tracking system in the 
invention of Chen in view of Rallison will accumulate an error in the measured 
directions, including the azimuth measurements. 

Claim 25 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent No. 5,765,561 to Chen et al. in view of U.S. Patent No. 6,369,952 to Rallison et 
al. and Fixsen et al. as applied to claim 23 and further in view of U.S. Patent No. 
5,531,227 to Schneider. 

Chen, Rallison, and Fixsen teach of the method of claim 25 except wherein a 
target image is created having a peripheral area around a predicted position in a picked- 
up image subjected to a rotational process on the basis of a rotational angle in a roll 
direction of said image pick-up device derived from said measured value. Chen et al. 
teaches in column 1 1, lines 3 - 1 1 , of a tracker system attacked to a video camera such 
that the position of the camera will be predetermined by the tracker system. Schneider 
teaches of obtaining images in a pre-procedural time and segmenting them into sub- 
objects that have defined boundaries, shapes, and positions within the overall image. 
Column 8, lines 36 - 52, describes segmenting the images while they are digitally 
analyzed to identify predefined fiducial markers by means of feature extraction, edge 
detection, region growing, boundary analysis, template matching, etc. Column 9, lines 4 
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- 10, states, "Particularly likely to be useful in the future are those statistical 
segmentation techniques that assign to each point a certain degree of probability as to 
whether or not it is a part of a given segmented object. That probability is based upon a 
variety of factors including pixel intensity and location with respect to other pixels of 
given qualities. Once probabilities of each pixel have been determined, assessments 
can be made of the pixels as a group, and segmentation can be achieved with improved 
accuracy." Additionally, column 10, lines 4-15, teaches of rotating captured images so 
as to align the images, according to the fiducial markers, to a corresponding position 
orientation. Thus, the position of the area to be segmented is determined by a 
statistical segmentation technique while the video camera is at a predetermined 
position. It would have been obvious to one having ordinary skill in the art at the time 
the invention was made to modify the invention of Chen et al. to include segmenting the 
captured virtual images so as to include a peripheral area around a predicted position of 
the index. One would have been motivated to make such a modification to the invention 
of Chen so that upon- performing template matching between virtual and real images, 
the speed of identifying marker matches is increased by bounding a search area within 
a smaller particular region of the overall image. It would have been further obvious to 
one having ordinary skill in the art at the time the invention was made to modify the 
invention of Chen to include subjecting the captured images to rotational process. One 
would have been motivated to make such a modification to the invention of Chen so that 
while performing template matching between a virtual and a real image, the orientations 
of the images may be aligned so as to properly line up their fiducial markers. 
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Conclusion 

The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 
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